A Critical Discussion of Alert Evaluations in the Context of Continuous Glucose Monitoring System Performance.

Many continuous glucose monitoring (CGM) systems provide functionality which alerts users of potentially unwanted glycemic conditions. These alerts can include glucose threshold alerts to call the user's attention to hypoglycemia or hyperglycemia, predictive alerts warning about impeding hypoglycemia or hyperglycemia, and rate-of-change alerts. A recent review identified 129 articles about CGM performance studies, of which approximately 25% contained alert evaluations. In some studies, real alerts were assessed; however, most of these studies retrospectively determined the timing of CGM alerts because not all CGM systems record alerts which necessitates manual documentation. In contrast to assessment of real alerts, retrospective determination allows assessment of a variety of alert settings for all three types of glycemic condition alerts. Based on the literature and the Clinical and Laboratory Standards Institute's POCT05 guideline, two common approaches to threshold alert evaluation were identified, one value-based and one episode-based approach. In this review, a critical discussion of the two approaches, including a post hoc analysis of clinical study data, indicates that the episode-based approach should be preferred over the value-based approach. For predictive alerts, fewer results were found in the literature, and retrospective determination of CGM alert timing is complicated by the prediction algorithms being proprietary information. Rate-of-change alert evaluations were not reported in the identified literature, and POCT05 does not contain recommendations for assessment. A possible approach is discussed including post hoc analysis of clinical study data. To conclude, CGM systems should record alerts, and the episode-based approach to alert evaluation should be preferred.

Benefits of Usability Evaluation in the Development Process of Diabetes Technologies Using the Example of a Continuous Glucose Monitoring System Prototype.

BACKGROUND: Usability engineering analyzes the interaction between the intended users and a device. Its implementation is mandatory for manufacturers to obtain regulatory approval for the European market. The aim of this evaluation was assessing the role of usability testing in the development process. For this purpose, a continuous glucose monitoring (CGM) device under development was investigated to determine whether it could be used safely and effectively by the intended users. METHODS: Conduct of the usability testing was based on the international standard IEC 62366-1. Medical device use of CGM-experienced and non-experienced users (n = 15 each) was observed without initial training in use scenarios containing 18 tasks. The success rate of task completion was determined and the System Usability Scale (SUS) score was calculated from a questionnaire. A prototype of the FiberSense CGM System (EyeSense GmbH, Großostheim, Germany), comprising of a single-use sensor and a reusable detector, was investigated. RESULTS: Most use errors made by both user groups were related to ease of handling of the reusable detectors. The SUS scores achieved in this study were below the pre-defined SUS score acceptance criterion of ≥68. The most frequently mentioned reason for use errors was an incomprehensible and non-chronological instructions for use (IFU). CONCLUSIONS: The evaluation provides valuable insights on how to improve usability of the prototype device and demonstrates the value of conducting structured usability testing prior to product finalization. The results reflected areas for improvement of the user interface, mainly by restructuring the IFU, provision of an additional leaflet, and device training prior to use.

Plasma Glucose Concentrations in Different Sampling Tubes Measured on Different Glucose Analysers.

INTRODUCTION: The German Diabetes Association recommends using sampling tubes with citrate and fluoride additives to diagnose diabetes by oral glucose tolerance test to inhibit glycolysis. The effect of different tubes on measurement results was assessed. MATERIALS AND METHODS: In a first study, an oral glucose tolerance test was performed on 41 participants without anamnestically known diabetes. Venous blood was sampled in two different tubes with citrate/fluoride additives from different manufacturers and one with only lithium-heparin additive. A second study with 42 participants was performed to verify the initial results with an adapted design, in which a third tube with citrate buffer was used, and glucose measurements were performed on two additional devices of another analyser model. Samples were centrifuged either immediately (<5 min incubation time) or after 20 min or 4 h. All glucose measurements were performed in plasma. Glucose concentrations in lithium-heparin tubes with<5 min incubation time served as baseline concentrations. RESULTS: In the first study, glucose concentrations in one of the citrate/fluoride tubes were similar to the baseline. In the other citrate/fluoride tube, markedly lower concentrations (approximately - 5 mg/dL (- 0.28 mmol/L)) were measured. This was reproduced in the verification study for the same analyser, but not with the other analyser model. Lithium-heparin tubes centrifuged after 20 and 240 min showed systematically lower glucose concentrations. CONCLUSIONS: The results confirm that glycolysis can be effectively inhibited in citrate/fluoride-containing sampling tubes. However, glucose measurement results of one analyser showed a relevant negative bias in tubes containing liquid citrate buffer.

A Proposal for the Clinical Characterization of Continuous Glucose Monitoring Trend Arrow Accuracy.

The assessment and characterization of trend accuracy, that is, the ability of a continuous glucose monitoring (CGM) system to correctly indicate the direction and rate of change (RoC) of glucose levels, has received comparatively little attention in the overall evaluation of CGM performance. As such, only few approaches that examine the trend accuracy have been put forward. In this article, we review existing approaches and propose the clinical trend concurrence analysis (CTCA) which is an adaptation of the conventional trend concurrence analysis. The CTCA is intended to directly evaluate the trend arrows displayed by the CGM systems by characterizing their agreement to suitably categorized comparator RoCs. Here, we call on manufactures of CGM systems to provide the displayed trend arrows for retrospective analysis. The CTCA classifies any deviations between the CGM trend and comparator RoC according to their risk for an adverse clinical event arising from a possibly erroneous treatment decision. For that, the existing rate error grid analysis and a specific set of trend arrow-based insulin dosing recommendations were used. The results of the CTCA are presented in an accessible graphical display and exemplified on data from three CGM systems. With this article, we hope to increase the awareness for the importance and challenges of assessing the accuracy of trend information displayed by CGM systems.

Investigation of the Effect of 70 Potential Interferents on Measurement Results of Two Blood Glucose Monitoring Systems.

BACKGROUND: Reliable blood glucose (BG) measurements are important for people with diabetes to manage their therapy as well as in point-of-care testing (POCT) performed by health care professionals to monitor BG of patients or even to diagnose diabetes. Among other factors, endogenous and exogenous substances present in blood samples can impact the measurement results. To ensure and prove that blood glucose monitoring systems (BGMSs) are robust in terms of potential interferents, manufacturers have to perform extensive evaluations. METHOD: An interference screening test was performed for three reagent system lots of a POCT system and of a BGMS for self-monitoring of BG. A paired-difference approach based on ISO 15197:2013 and CLSI guideline EP07 was used with venous whole blood samples at two different glucose concentrations. Seventy potential interferents expected to be common in people with diabetes were evaluated. RESULTS: The interference effects were determined as normalized biases between test samples and corresponding control samples. For 69 of the 70 investigated potential interferents, both systems met the predefined acceptance criteria, with the normalized biases falling within ±10 mg/dL or ±10% at glucose concentrations ≤100 mg/dL or >100 mg/dL, respectively, for each of the three evaluated reagent system lots. CONCLUSIONS: The BGMS investigated in this study were found to be robust with respect to the 70 evaluated potential interferents. Interference effects were observed only for N-Acetyl-L-cysteine. Extensive evaluations of potential interfering factors can make an important contribution to ensure reliability of BGMS.

Comparator Data Characteristics and Testing Procedures for the Clinical Performance Evaluation of Continuous Glucose Monitoring Systems.

Comparing the performance of different continuous glucose monitoring (CGM) systems is challenging due to the lack of comprehensive guidelines for clinical study design. In particular, the absence of concise requirements for the distribution of comparator (reference) blood glucose (BG) concentrations and their rate of change (RoC) that are used to evaluate CGM performance, impairs comparability. For this article, several experts in the field of CGM performance testing have collaborated to propose characteristics of the distribution of comparator measurements that should be collected during CGM performance testing. Specifically, it is proposed that at least 7.5% of comparator BG concentrations are <70 mg/dL (3.9 mmol/L) and >300 mg/dL (16.7 mmol/L), respectively, and that at least 7.5% of BG-RoC combinations indicate fast BG changes with impending hypo- or hyperglycemia, respectively. These proposed characteristics of the comparator data can facilitate the harmonization of testing conditions across different studies and CGM systems and ensure that the most relevant scenarios representing real-life situations are established during performance testing. In addition, a study protocol and testing procedure for the manipulation of glucose levels are suggested that enable the collection of comparator data with these characteristics. This work is an important step toward establishing a future standard for the performance evaluation of CGM systems.

Effect of Arterialization on Venous Blood Glucose Concentrations and Implications for Observed Continuous Glucose Monitoring Accuracy.

Background: Heating of the arm and/or hand ("arterialization") is sometimes used in continuous glucose monitoring (CGM) performance studies with the reported aim of reducing differences between venous and capillary glucose concentrations. In this study, the effect of heating on venous glucose concentrations and CGM accuracy was investigated. Methods: A heating pad set to 50°C (122°F) was used with 20 participants to heat either the dominant or nondominant arm and hand. Venous and capillary samples were obtained every 15 min on both arms throughout each of three 6-h glucose challenges. CGM sensors were worn on each upper arm for each of the three visits. Results: Heating of the arm led to a median increase in venous glucose concentrations of +1.4%. No similar effect on capillary concentrations was observed. As a result, the median capillary to venous difference decreased from +5.9% in the nonheated arm to +4.2% in the heated arm. CGM accuracy observed in this study was affected by the selection of heated venous, nonheated venous, or capillary glucose concentrations as comparator data. The heating effect was more pronounced with rapidly decreasing glucose concentrations. Temperatures on the skin did not exceed 40°C (104°F). No adverse events or protocol deviations were associated with the use of the heating pad. Conclusions: Heating of the arm led to a small increase in venous glucose concentrations, but venous concentrations did not reach the level of capillary glucose concentrations. CGM accuracy observed in this study varied depending on the selected comparator data. This study was registered at the German Clinical Trials Register (DRKS00031197).

Evaluation of System Accuracy, Precision, Hematocrit Influence, and User Performance of Two Blood Glucose Monitoring Systems Based on ISO 15197:2013/EN ISO 15197:2015.

INTRODUCTION: Sufficiently high analytical quality of blood glucose monitoring systems (BGMS) is a prerequisite for efficient diabetes therapy. In this study we assessed system accuracy, measurement repeatability, intermediate measurement precision, user performance, and the influence of hematocrit on two CE-marked blood glucose monitoring systems. For one BGMS, measurement accuracy using venous samples was additionally investigated. METHODS: Study procedures were based on the International Organization of Standardization (ISO) 15197:2013/EN ISO 15197:2015 ("ISO 15197"). User performance included data from 100 subjects who used one test strip lot, whereas for all other analyses three different reagent system lots were used. For system accuracy assessment, 100 capillary samples were measured in duplicate with each of three reagent system lots per system, resulting in 600 results per system. RESULTS: CareSens S Fit and CareSens H Beat both fulfilled the ISO 15197 accuracy criteria with 97.5-100% of each test strip lot's results falling within ± 15 mg/dL or ± 15% of the results of the comparison method and 100% of results in consensus error grid (CEG) zone A for all three lots. User performance evaluation revealed sufficient accuracy in the hands of lay users although some handling errors were documented by study staff. Assessment of measurement repeatability and intermediate measurement precision is given by standard deviation (SD) (glucose levels < 100 mg/dL) and by coefficient of variation (CV) (glucose concentrations ≥ 100 mg/dL). SD was ≤ 4.1 mg/dL and CV ≤ 4.2% for measurement repeatability and SD was ≤ 2.2 mg/dL and CV ≤ 2.6% for intermediate measurement precision. In case of hematocrit influence, both BGMS complied with all three tested lots with the defined criteria. CONCLUSION: Both BGMS analyzed in this study fulfilled the required accuracy criteria of ISO 15197. They showed high precision, good performance in the hands of lay users, and the influence of hematocrit was acceptable in the labeled range.

Performance Assessment of Three Continuous Glucose Monitoring Systems in Adults With Type 1 Diabetes.

BACKGROUND: FIND, the global alliance for diagnostics, identified the nonmarket-approved continuous glucose monitoring (CGM) system, FiberSense system (FBS), as a potential device for use in low- and middle-income countries. Together with two market-approved, factory-calibrated CGM systems, namely, the FreeStyle Libre 2 (FL2) and the GlucoRx AiDEX (ADX), the FBS was subjected to a clinical performance evaluation. METHODS: Thirty adult participants with type 1 diabetes were enrolled. The study was mainly conducted at home, with three in-clinic sessions conducted over the study period of 28 days. Comparator measurements were collected from capillary samples, using a high-quality blood glucose monitoring system. RESULTS: Data from 31, 70, and 78 sensors of FBS, FL2, and ADX, respectively, were included in the performance analysis. The mean absolute relative differences between CGM and comparator data for FBS, FL2, and ADX were 14.7%, 9.2%, and 21.9%, and relative biases were -2.1%, -2.5%, and -18.5%, respectively. Analysis of individual sensor accuracy revealed low, moderate, and high sensor-to-sensor variability for FBS, FL2, and ADX, respectively. Sensor survival probabilities until the end of sensor life were 47.2% for FBS (28 days), 71.3% for FL2 (14 days), and 48.4% for ADX (14 days). CONCLUSIONS: The results of FBS were encouraging enough to conduct further performance and usability evaluations in a low- and middle-income country. The results of FL2 mainly agreed with existing studies, whereas ADX showed substantial deviations from previously reported results.

Clinical Performance Evaluation of Continuous Glucose Monitoring Systems: A Scoping Review and Recommendations for Reporting.

The use of different approaches for design and results presentation of studies for the clinical performance evaluation of continuous glucose monitoring (CGM) systems has long been recognized as a major challenge in comparing their results. However, a comprehensive characterization of the variability in study designs is currently unavailable. This article presents a scoping review of clinical CGM performance evaluations published between 2002 and 2022. Specifically, this review quantifies the prevalence of numerous options associated with various aspects of study design, including subject population, comparator (reference) method selection, testing procedures, and statistical accuracy evaluation. We found that there is a large variability in nearly all of those aspects and, in particular, in the characteristics of the comparator measurements. Furthermore, these characteristics as well as other crucial aspects of study design are often not reported in sufficient detail to allow an informed interpretation of study results. We therefore provide recommendations for reporting the general study design, CGM system use, comparator measurement approach, testing procedures, and data analysis/statistical performance evaluation. Additionally, this review aims to serve as a foundation for the development of a standardized CGM performance evaluation procedure, thereby supporting the goals and objectives of the Working Group on CGM established by the Scientific Division of the International Federation of Clinical Chemistry and Laboratory Medicine.

Comment on Hochfellner et al. Accuracy Assessment of the GlucoMen® Day CGM System in Individuals with Type 1 Diabetes: A Pilot Study. Biosensors 2022, 12, 106.

In their recent article entitled "Accuracy Assessment of the GlucoMen® Day CGM System in Individuals with Type 1 Diabetes: A Pilot Study" [...].

A Statistical Approach for Assessing the Compliance of Integrated Continuous Glucose Monitoring Systems with Food and Drug Administration Accuracy Requirements.

To assess the compliance of "integrated" continuous glucose monitoring (CGM) systems with U.S. Food and Drug Administration requirements, the calculation of confidence intervals (CIs) on agreement rates (ARs), that is, the percentage of CGM measurements lying within a certain deviation of a comparator method, is stipulated. However, despite the existence of numerous approaches that could yield different results, a specific procedure for calculating CIs is not described anywhere. This report, therefore, proposes a suitable statistical procedure to allow transparency and comparability between CGM systems. Three existing methods were applied to six data sets from different CGM performance studies. The results indicate that a bootstrap-based method that accounts for the clustered structure of CGM data is reliable and robust. We thus recommend its use for the estimation of CIs of ARs. A software implementation of the proposed method is freely available (https://github.com/IfDTUlm/CGM_Performance_Assessment).

Impact of Blood Glucose Monitoring System Accuracy on Clinical Decision Making for Diabetes Management.

BACKGROUND: The accuracy of blood glucose monitoring systems (BGMS) is crucial for the safe and effective management of diabetes mellitus. Despite standardization of accuracy assessment procedures and requirements, various studies have shown that the accuracy of BGMS on the market can vary considerably. This article therefore provides health care professionals and users with an intuitive illustration of the impact of BGMS accuracy on clinical decision making. MATERIAL AND METHODS: Several hypothetical patient scenarios based on blood glucose (BG) levels in the low, normal, and high BG range are devised. Using data from a recent BGMS accuracy study, a method for calculating the expected range of BG readings from four examined BGMS at the selected BG levels is introduced. Based on these ranges, it is illustrated how clinical decisions and subsequent outcomes of the hypothetical patients are affected by the expected inaccuracies of the BGMS. RESULTS: The range of expected BGMS readings for the same true BG level can vary considerably between different BGMS. The discussion of hypothetical patient scenarios revealed that the use of some BGMS could be associated with an increased risk of adverse events such as failure to detect hypoglycemia, driving with an unsafe BG level, delay of treatment intervention in diabetes during pregnancy, or the failure to prevent diabetic ketoacidosis. CONCLUSIONS: This article can support both health care professionals and patients to understand the impact of BGMS accuracy in a relatable, clinical context. Furthermore, it is suggested that current accuracy requirements might be insufficient for the prevention of adverse clinical outcomes in certain circumstances.

Continuous Glucose Deviation Interval and Variability Analysis (CG-DIVA): A Novel Approach for the Statistical Accuracy Assessment of Continuous Glucose Monitoring Systems.

BACKGROUND: The accuracy of continuous glucose monitoring (CGM) systems is crucial for the management of glucose levels in individuals with diabetes mellitus. However, the discussion of CGM accuracy is challenged by an abundance of parameters and assessment methods. The aim of this article is to introduce the Continuous Glucose Deviation Interval and Variability Analysis (CG-DIVA), a new approach for a comprehensive characterization of CGM point accuracy which is based on the U.S. Food and Drug Administration requirements for "integrated" CGM systems. METHODS: The statistical concept of tolerance intervals and data from two approved CGM systems was used to illustrate the CG-DIVA. RESULTS: The CG-DIVA characterizes the expected range of deviations of the CGM system from a comparison method in different glucose concentration ranges and the variability of accuracy within and between sensors. The results of the CG-DIVA are visualized in an intuitive and straightforward graphical presentation. Compared with conventional accuracy characterizations, the CG-DIVA infers the expected accuracy of a CGM system and highlights important differences between CGM systems. Furthermore, it provides information on the incidence of large errors which are of particular clinical relevance. A software implementation of the CG-DIVA is freely available (https://github.com/IfDTUlm/CGM_Performance_Assessment). CONCLUSIONS: We argue that the CG-DIVA can simplify the discussion and comparison of CGM accuracy and could replace the high number of conventional approaches. Future adaptations of the approach could thus become a putative standard for the accuracy characterization of CGM systems and serve as the basis for the definition of future CGM performance requirements.

User Performance Evaluation and System Accuracy Assessment of Four Blood Glucose Monitoring Systems With Color Coding of Measurement Results.

BACKGROUND: Blood glucose monitoring systems (BGMSs) are a cornerstone in diabetes management. They have to provide sufficiently accurate results in the hands of lay users, particularly in insulin-treated patients. The aim of this study was user performance evaluation and system accuracy assessment of four BGMSs with color coding of results. METHODS: Study procedures were based on ISO 15197:2013. User performance evaluation included data from 100 participants, each of whom used every system with one reagent lot. Study personnel observed user techniques. For the system accuracy assessment, 100 capillary samples were obtained for measurement in duplicate with each of three reagent system lots per system, resulting in 600 results per system. RESULTS: All assessed BGMSs exhibited a sufficient level of accuracy, with small differences between them. In the user performance evaluation, study personnel observed the smallest total number of user errors with Contour Next (Ascensia), followed by Accu-Chek Instant (Roche), Medisafe Fit Smile (Terumo), and OneTouch Ultra Plus Reflect (LifeScan). Approximately 90% of participants stated that a consistent color scheme, eg, for low blood glucose (BG) values, should be used across all BGMSs. There was no clear preference among the four tested BGMSs regarding the specific way of displaying color coding. CONCLUSIONS: The four BGMSs assessed in this study showed only small differences in an overall sufficient level of accuracy. User handling errors, as observed by study personnel, differed between the systems.

Improving the Bias of Comparator Methods in Analytical Performance Assessments Through Recalibration.

BACKGROUND: In analytical performance studies, the choice of comparator method plays an important role, as studies have shown that there exist relevant systematic differences (bias) between laboratory analyzers. The feasibility of retrospective recalibration of measurement results through comparison with methods or materials of higher metrological order to minimize bias was therefore assessed. METHOD: Existing data from performance studies of continuous and blood glucose monitoring systems were retrospectively analyzed. Comparison with a higher-order method was performed for two different data sets. In both cases, subject samples were measured, and a subset was also measured on a higher-order method. Recalibration based on higher-order materials (standard reference material [SRM]) was conducted for two different data sets containing results from SRM and subject samples. Linear regression analysis was performed for each device separately. Resulting equations were applied to the respective complete data set of subject samples. Bias between devices in a data set across all subject samples was assessed before and after recalibration. RESULTS: Bias between devices was reduced from -3.6% to +0.6% in one data set and from +11.0% to +0.3% in the other by recalibration based on higher-order method. Using higher-order materials, bias was also reduced by recalibration, but mixed results were found: Bias was reduced from -3.1% to -0.1% in one data set and from -4.3% to -2.7% in the other. CONCLUSIONS: Recalibration did lead to a decrease in bias and thus can reduce the impact of the choice of comparator method. The procedure should be verified in a prospectively designed setting.

Continuous Glucose Profiles in Healthy People With Fixed Meal Times and Under Everyday Life Conditions.

BACKGROUND: The increased use of continuous glucose monitoring (CGM) and automated insulin delivery systems raises the question about therapeutic targets for glucose profiles in people with diabetes. This study aimed to assess averaged pre- and postprandial glucose profiles in people without diabetes to provide guidance for normal glucose patterns in clinical practice. For that, number and timing of meal intake were predefined. MATERIAL AND METHODS: To assess glucose traces in 36 participants without diabetes (mean age = 23.7 ± 5.7 years), CGM was performed for up to 14 days, starting with a run-in phase (first 3 days, excluded from analysis) followed by 4 days with fixed meal times at 8:00 am, 1:00 pm, and 6:00 pm and the remaining 7 days spent under everyday life conditions. Data from two simultaneously worn CGM sensors were averaged and adjusted to capillary plasma-equivalent glucose values. Glucose data were evaluated through descriptive statistics. RESULTS: Median glucose concentration on days with fixed meal times and under everyday life conditions was 95.0 mg/dL (91.6-99.1 mg/dL, interquartile range) and 98.1 mg/dL (93.7-100.8 mg/dL), respectively. On days with fixed meal times, mean premeal glucose was 92.8 ± 9.4 mg/dL, and mean peak postmeal glucose was 143.3 ± 23.5 mg/dL. CONCLUSIONS: By defining the time of meal intake, a clear pattern of distinct postprandial glucose excursions in participants without diabetes could be demonstrated and analyzed. The presented glucose profiles might be helpful as an estimate for adequate clinical targets in people with diabetes.